One of the most difficult computer graphics challenges is rendering realistic human faces. Humans are incredibly adept at interpreting facial appearance. For example, we can easily distinguish if a person is happy, tired, hot, angry, excited, or sick. Although a lot of effort has been devoted to face modeling in computer graphics, no synthetic face model to date achieves the level of expressiveness of human faces.
Realism requires an accurate model for the skin reflectance of human faces. Skin reflectance varies for different people, e.g., due to age, race, gender, health, etc. Skin reflectance even varies for the same person throughout the course of a day, e.g., hot vs. cold skin, or dry vs. wet.
Properties of human skin have been measured and studied extensively in biomedical, cosmetic, and computer vision fields.
Analytic Skin Reflectance Models
Analytic reflectance models are frequently used because of their computational efficiency, Hanrahan and Krueger, “Reflection from layered surfaces due to subsurface scattering,” Computer Graphics, SIGGRAPH '93 Proceedings, 165-174, 1993. They model a single scattering of light in skin composed of multiple smoothly bounded internal layers. That model was extended by adding an oil layer at the skin surface, Ng and Li, “A multi-layered reflection model of natural human skin,” Computer Graphics International, 249-256, 2001.
Another model applies an analytic approximation to multiple subsurface scattering in skin with a rough surface, Stam, “An illumination model for a skin layer bounded by rough surfaces,” Proceedings of the 12th Eurographics Workshop on Rendering Techniques, Springer, 39-52, 2001.
A physically-based multi-layer model for image synthesis uses biologically meaningful parameters, Krishnaswamy and Baranoski, “A biophysically-based spectral model of light interaction with human skin,” Computer Graphics Forum 23, 331-340, September 2004.
Several skin modeling approaches use analytic bidirectional reflectance distribution functions (BRDFs), Blanz and Vetter, “A morphable model for the synthesis of 3D faces,” Computer Graphics 33, Annual Conference Series, 187-194, 1999; Debevec et al., “Acquiring the reflectance field of a human face,” Computer Graphics, SIGGRAPH 2000 Proceedings, 145-156, 2000; Haro et al., “Real-time, photo-realistic, physically based rendering of fine scale human skin structure,” Proceedings of the 12th Eurographics Workshop on Rendering Techniques, 53-62, 2001; Paris et al., “Lightweight face relighting,” Proceedings of Pacific Graphics, 41-50, 2003; Tsumura et al., “Image-based skin color and texture analysis/synthesis by extracting hemoglobin and melanin information in the skin,” ACM Transactions on Graphics, SIGGRAPH 2003, 22, 3, 770-779, 2003; and Fuchs et al., “Reflectance from images: A model-based approach for human faces,” Research Report MPI-I-2005-4-001, Max-Planck-Institut fur Informatik, 2005.
The BRDF parameters can be estimated from reflectance measurements using non-linear optimization. Although a BRDF describes local light transport at each surface point, it ignores subsurface scattering, which is largely responsible for the appearance of skin.
Another analytic model uses bi-directional surface-scattering reflectance distribution function (BSSRDF), Jensen et al., “A rapid hierarchical rendering technique for translucent materials,” Computer Graphics, SIGGRAPH 2002 Proceedings, 576-581, 2002; and Jensen et al., “A practical model for subsurface light transport,” Computer Graphics, SIGGRAPH 2001 Proceedings, 511-518, 2001.
The BSSRDF describes the full effect that incident light has on reflected light from a surface patch around a point. The BSSRDF is eight-dimensional, assuming a two-dimensional parameterization of the surface. However, dense sampling of an eight-dimensional function is difficult using conventional techniques.
Non-Parametric Skin Reflectance Models
Another method estimates a non-parametric BRDF of a human face by combining reflectance samples from different points on the surface, Marschner et al., “Image-based BRDF measurement including human skin,” Proceedings of the 10th Eurographics Workshop on Rendering, 139-152, 1999. They later extended this work by adding a detailed albedo texture, Marschner et al., “Modeling and rendering for realistic facial animation,” 11th Eurographics Workshop on Rendering, 231-242, 2000. They observed that the BRDF of skin exhibits strong forward scattering at grazing angles that is uncorrelated with a specular direction.
A data-driven BRDF model is described by Matusik, et al., “A data-driven reflectance model,” ACM Transactions on Graphics, SIGGRAPH 2003, 22, 3 (July), 759-770, 2003. That model estimates a non-parametric surface BRDF at each surface point. This introduces less error than imposing the behavior of a particular analytic BRDF model. More important, the data-driven BRDF model does not require non-linear optimization and leads to a more robust fitting procedure.
Image-Based Face Modeling
Image-based models have provided realistic representations for human faces. Image-based models easily represent effects such as self-shadowing, inter-reflections, and subsurface scattering, Pighin et al., “Synthesizing realistic facial expressions from photographs,” Computer Graphics, vol. 32 of SIGGRAPH 98 Proceedings, 75-84, 1998.
Other image-based models allow variations in lighting, Georghiades et al., “Illumination-based image synthesis: Creating novel images of human faces under differing pose and lighting,” IEEE Workshop on Multi-View Modeling and Analysis of Visual Scenes, 47-54, 1999; Debevec et al., “Acquiring the reflectance field of a human face,” Computer Graphics, SIGGRAPH 2000 Proceedings, 145-156, 2000; and viewpoint, and expression, Hawkins et al., “Animatable facial reflectance fields,” Rendering Techniques '04, Proceedings of the Second Eurographics Symposium on Rendering, 2004; and Cula et al., “Skin texture modeling,” International Journal of Computer Vision 62, 1-2 (April-May), 97-119, 2005.
However, the memory requirement for image-based models is large. The measurement procedures are inefficient and assume non-local low-frequency lighting. Pure image-based representations are also inherently difficult to edit and modify.
Another model combines an image-based model, an analytic surface BRDF, and an approximation of subsurface scattering to create highly realistic face images for the movie ‘Matrix Reloaded’, Borshukov and Lewis, “Realistic human face rendering for the Matrix Reloaded,” SIGGRAPH 2003 Conference Abstracts and Applications (Sketch), 2003.
A variant of that method for real-time skin rendering uses graphics hardware, Sander et al., “Real-time skin rendering on graphics hardware,” SIGGRAPH 2004, Sketch, 2004.
U.S. Patent Application 20040150642, “Method for digitally rendering skin or like materials,” filed by Borshukov and Lewis on Aug. 5, 2004, describes a method for rendering skin tissue. That method applies a blurring process to a two-dimensional light map.
Another image-based uses independent component analysis (ICA) to decompose images of faces into layers, e.g., melanin and hemoglobin, Tsumura et al., 2003. That method is capable of re-synthesizing new images of faces while adding effects such as tanning and aging.
A realistic skin reflectance model should be able to accommodate a wide variation of face appearance. The model should also allow a graphic artist to change the appearance of skin based on easy to interpret parameters, e.g., race, gender, or age. The model needs to incorporate measurements of real faces for rendering virtual duplicates. Images generated from the model, ideally in real-time, need to appear photorealistic from arbitrary viewpoints. And the model should allow easy modification and transfer of skin appearance.